Feature Selection using Distributed Ensemble Classifiers for Very Large Datasets

Datasets are becoming larger and there is an acute need to use data mining techniques to exploit the available data. The increasing size of the datasets poses a challenge to the data miners, which can be solved using two approaches – high speed computing systems, and pre-processing techniques. In this paper, we propose a solution combining the above two techniques using a distributed feature selection method to address the challenge of mining very large datasets. The dependencies of the features of a dataset are computed by an ensemble of classifiers in a distributed computing environment called Hadoop, leading to faster processing and increased reliability. A subset of features is selected based on the averaged estimated dependency vector. The proposed model is verified with different data sets and the validated results are presented. Keywords-data mining; feature selection; large datasets; data preprocessing; mapreduce